Skip to content

feat(terminal): preview IME composition text on iOS Safari - #499

Open
aakhter wants to merge 3 commits into
Ark0N:masterfrom
aakhter:pr/mobile-ime-preview
Open

aakhter wants to merge 3 commits into
Ark0N:masterfrom
aakhter:pr/mobile-ime-preview

Conversation

@aakhter

@aakhter aakhter commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

What

On iOS Safari, text being composed by an IME (Japanese, Chinese, Korean, and also the predictive/autocorrect composition on English keyboards) is not visible in the terminal until it commits, so users type blind. This adds mobile-ime-preview.js, a visual-only controller that paints the in-progress composition in the xterm helper layer (at the same position as the helper textarea), and holds a committed chunk on screen until something authoritative shows it: the local-echo overlay, parsed terminal output that arrived after the commit, or a 2s fallback.

How

  • src/web/public/mobile-ime-preview.js: the controller (MobileImePreview.create() / isIosWebKitTouch()). It only mounts on iOS WebKit touch devices; everywhere else _initMobileImePreview() returns early and no DOM is added.
  • terminal-ui.js: init/destroy around terminal.open(), a reset on session switch, and two small hooks:
    • handleTerminalData: if the chunk is the IME's commit and local echo is on, it goes to the overlay (which then displays it in place of the preview). If the overlay cannot take it, the text is sent directly rather than dropped.
    • batchTerminalWrite / flush: an output arrival counter, so the preview only clears on output that arrived after the commit and was fully parsed. Output queued before the commit, a split chunk, or another session's output never clears it early.
  • Registered like terminal-keycode229-recovery.js: script tag, build minify step, HASHABLE entry (so sw.js precaches it), three CSS rules, and the CLAUDE.md load order.

Every code path is wrapped so a preview failure can never block or drop input.

Testing

  • test/mobile-ime-preview.test.ts (34) covers the controller. test/mobile-ime-preview-structure.test.ts (31) covers the wiring, script order, build/hash registration, CSS and the terminal-ui.js hooks.
  • Mutation-checked: removing the commit marker, the split-chunk guard, the output-order compare, the overlay handoff, the session-switch reset, the init call, the script tag or the HASHABLE entry each fails at least one test.
  • Neighbouring suites pass unchanged: local-echo-codex-gating, terminal-touch-tap, terminal-flush-budget, terminal-keycode229-recovery, input-cjk, terminal-buffer-flush, app-settings-structure, sw-precache-manifest.
  • tsc, lint, check:frontend-syntax, format:check and check:public-assets are clean. npm run build emits the hashed asset, and both index.html and sw.js reference it.
  • Browser check (chromium, iPhone UA + touch): the controller mounts, and a synthetic composition renders its text in the preview. On desktop the module is inert with no preview node. No page errors in either.

Reviewer notes

  • Not validated on a physical iPhone from this branch. The composition was driven with synthetic compositionstart/compositionupdate/input events.
  • Codex's predictive-echo path is not wired to the preview. For codex sessions the preview clears when the echoed output is parsed, or on the 2s fallback.

WebKit on iOS does not show text being composed by an IME inside the
terminal, so users type blind until it commits. Add mobile-ime-preview.js,
a visual-only controller that renders the composition in the xterm helper
layer and holds a committed chunk until local echo, parsed terminal output
or a 2s fallback shows it. Wire it into terminal-ui.js, the script order,
the build minify/hash lists and styles, with unit and wiring tests.
@Ark0N

Ark0N commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Thanks for this, @aakhter. This adds an in-terminal preview of IME composition text on iOS, so people typing Japanese, Chinese or Korean can see what they are composing before it commits, and it stays fully inert everywhere else. The wiring is careful: the output-sequence guard against pre-commit output, the split-chunk guard, the session-switch reset, the codex byte-identity and the build, HASHABLE and load-order registration are all right, and the full test gate is green here (434 files).

A few things before merge:

1. The keydown finalization runs after xterm has already finalized (src/web/public/mobile-ime-preview.js:156). xterm 6.0 registers its textarea keydown listener with capture: true inside terminal.open() (CoreBrowserTerminal.ts:379), so it runs before the controller's capture listener. For any keyCode other than 229/20/16/17/18, xterm's CompositionHelper.keydown finalizes and emits the commit synchronously through onData while the controller is still composing, so consumeTerminalData returns false. The controller's own keydown handler then enters awaitingCommit and paints the old composition as provisional text, with no timer, until the next printable keystroke (which is then adopted as the IME commit), a blur or a session switch. I reproduced it with real xterm 6.0 in Chromium: after compositionstart, compositionupdate 確定 and an Enter keydown, onData delivers 確定 and \r with consumed: false, and the controller ends at awaitingCommit: true, timerPending: false with { text: '確定', phase: 'provisional' } on screen. The opposite case diverges too: a keydown with keyCode 229 and isComposing: false stops the controller tracking while xterm keeps composing, so the preview freezes. The test at test/mobile-ime-preview.test.ts:231 registers the xterm stand-in as a bubble listener, which is why it passes. Ask: finalize only where CompositionHelper.keydown does, observe the keydown before xterm (a capture listener on terminal.element fires before any listener on the textarea), and register the test's xterm stand-in with capture: true ahead of the controller.

2. Bound the awaitingCommit state (src/web/public/mobile-ime-preview.js:138). Only the committed phase has the 2 s fallback. A composition whose commit never reaches onData (the user deletes the whole composition, so xterm emits nothing) leaves the controller waiting, and the next unrelated keystroke or single-line paste is painted as a committed IME chunk. Arming the same TTL on entering awaitingCommit fixes this and backs up item 1.

3. A run on a real iPhone. Every line of this executes only on iOS WebKit, and the description notes it was driven with synthetic events in Chromium. xterm's own composition view (.composition-view.active) does render the compositionupdate text, so a short before/after screen recording on iOS Safari would settle the premise and the event shapes items 1 and 2 depend on: Japanese kana and Chinese pinyin, local echo on (a Claude session) and off (a shell session), including Return pressed mid-composition. If you have no iPhone at hand, say so and I will run it on mine.

4. pagehide destroys the controller for good (src/web/public/terminal-ui.js:374). initTerminal() runs once per page load, and iOS Safari restores pages from the back-forward cache (notification-manager.js already handles pageshow with persisted), so after that round trip the preview stays off until a full reload. Drop the listener (a real unload frees everything anyway) or call reset() there instead.

Smaller things I can also fold in at merge time if you prefer:

  • src/web/public/styles.css:463: the preview span has no background, so it overprints whatever sits at the cursor (Claude's dim composer placeholder, for one). xterm's composition view is opaque for this reason; setting the terminal theme background next to the foreground in syncPreviewTypography matches it.
  • src/web/public/terminal-ui.js:418: _transferMobileImeCommitToLocalEcho repeats the printable-char branch's addChar/appendText choice. Letting that branch run and calling completeCommit({ predicted: true }) afterwards keeps one code path and drops the send-on-throw fallback, which would double-send if the overlay ever threw after appending.
  • CLAUDE.md: the load-order entry is right. A sentence next to the keycode229 paragraph (an iOS IME commit is routed into the overlay via _transferMobileImeCommitToLocalEcho, and the preview clears only on output parsed after the commit), plus the new z-index 6 layer in the Z-index list, saves the next person a search.

Once 1 to 4 are addressed and it has been seen working on an iPhone, this is ready to merge.

- Observe keydown in the capture phase on terminal.element, an ancestor of
  the helper textarea, so the controller sees it before xterm's own capture
  listener finalizes the composition and emits the commit through onData.
  Finalize on exactly the keys CompositionHelper.keydown does (every keyCode
  except 20/229/16/17/18), ignoring isComposing and key as xterm does.
- Bound awaitingCommit with the same 2 s fallback as the committed phase, so
  a composition whose commit never reaches onData cannot turn the next
  unrelated keystroke or paste into an IME commit.
- pagehide resets the controller instead of destroying it, so a back-forward
  cache restore keeps the preview working.
- Give the preview an opaque background from the terminal theme.
- Route an IME commit through the ordinary printable/paste local echo branch
  and complete the commit afterwards; drop the send-on-throw fallback.
- Pin the event order with an xterm stand-in registered in the capture phase
  ahead of the controller, and against real xterm in a browser test.
- CLAUDE.md: note the IME commit routing and the z-index 6 preview layer.
@aakhter

aakhter commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review, and for reproducing item 1 against real xterm.

  1. Fixed. The controller now listens for keydown in the capture phase on terminal.element, so it runs before xterm's textarea listener. It finalizes on exactly the keys CompositionHelper.keydown does in 6.0.0 (anything except 20/229/16/17/18 while composing), ignoring isComposing and key, just as xterm does. The unit test's xterm stand-in is now a capture listener on the textarea registered ahead of the controller, which is what the old test got wrong. Your Enter scenario and the 229 with isComposing: false case both fail against the previous code and pass now. I also added test/mobile-ime-preview.browser.test.ts (registered in BROWSER_TEST_GLOBS), which drives real xterm 6.0 in Chromium through the same cases plus a deleted composition followed by a paste. Pointing the controller back at the textarea fails it.
  2. Done. Entering awaitingCommit arms the same 2 s bound as the committed phase, tested with fake timers.
  3. I don't have an iPhone, so I'd gratefully take you up on the offer to run it on yours. The cases you listed (kana, pinyin, local echo on in a Claude session and off in a shell, Return mid-composition) are the ones I'd most like to see too.
  4. pagehide now calls reset() instead of destroying the controller, so a back-forward cache restore keeps it working.

Small ones, all done: the preview takes the terminal theme background alongside the foreground; the commit now goes through the normal printable/paste branch with completeCommit({ predicted: true }) afterwards, so the send-on-throw fallback is gone; and CLAUDE.md has the routing note plus the z-index 6 layer.

@Ark0N

Ark0N commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Thanks for the quick turnaround, @aakhter. This PR paints the text an IME is composing inside the terminal on iOS Safari, and the second commit covers everything from the first round: the capture-phase keydown on terminal.element matches CompositionHelper.keydown, awaitingCommit is bounded, pagehide resets instead of destroying, the preview has an opaque background, and the commit goes through the normal printable/paste branch. The new browser test against real xterm 6.0 passes here, and so does the full gate.

One thing needs a change before merge:

Later compositions in a prompt are drawn under the local-echo overlay (src/web/public/terminal-ui.js:311, src/web/public/styles.css:463). Local echo is on by default for Claude sessions on phones, so committed text sits in the overlay's pendingText and does not reach the PTY until Enter. The PTY cursor therefore stays at the prompt start, and that is where --xterm-helper-left/top places the preview. Two stacking facts decide the rest:

  • The overlay is a z-index: 7 layer in .xterm-screen, and its first line div is opaque from the prompt column to the right edge.
  • The preview's z-index: 6 only counts inside .xterm-helpers, which is its own z-index: 5 stacking context.

I reproduced it in Chromium with real xterm 6.0, the real overlay bundle and styles.css. I composed 今日は on an empty prompt (visible) and let it commit into the overlay. Then I composed 天気, and the overlay's 今 covered the whole second composition. So in a Claude session only the first word of each prompt gets a preview.

The ask: when the overlay has text, draw the composition after its pending text and above it. One option is a small addition to the overlay itself: a setComposition(text) in packages/xterm-zerolag-input/src/ that renders the provisional text as a styled tail after pendingText, reusing its wrapping and grow-upward layout. That keeps overlay behavior in its single source. The other option is to keep the span, place it at prompt column + cell width of pendingText, and mount it in .xterm-screen above z-index 7. Either way, please add a browser test that composes a second segment with text already pending. Every current test composes on an empty prompt, and that is how this got through.

One small thing, fine to fold in with the fix, or I can do it at merge time:

  • CLAUDE.md:375 lists the preview at z-index 6, but that value only counts inside .xterm-helpers (z-index 5), and docs/architecture-invariants.md:952 has no entry for it. Once the preview's home is settled, state its effective layer in both places.

Once that lands I will run it on my iPhone as promised: kana and pinyin, local echo on in a Claude session and off in a shell, and Return mid-composition. If that looks right, this is ready to merge.

With local echo on, committed text sits in the LocalEchoOverlay and does not
reach the PTY before Enter, so the PTY cursor that places the preview span
stays at the prompt start. The span's z-index 6 only counts inside
.xterm-helpers (its own z-index 5 stacking context), and the overlay is a
z-index 7 layer whose first line is opaque from the prompt column, so every
composition after the first one in a prompt was drawn under the overlay.

- xterm-zerolag-input: add setComposition(text) and a composition getter.
  The overlay draws the composition as an underlined, aria-hidden tail after
  its pending text, through the same wrapping and grow-upward layout. It is
  never part of pendingText, hasPending or anything sent; clear() and
  removeChar() drop it, and rerender()/refreshFont() keep it.
- terminal-ui.js: while local echo shows typed text (on, and not handed back
  to PTY echo by a nav key), render and clear the preview through
  setComposition. The helper span stays for local echo off, and as the
  fallback when the overlay cannot place the text (no prompt found).
- Browser test against real xterm 6, the overlay bundled from its source
  and styles.css: a second composition after pending text is the topmost
  element after that text, and the commit lands in the overlay once. Unit
  tests for setComposition in the package and for the routing in the
  structure test.
- CLAUDE.md and architecture-invariants: state the preview's effective layer.
@aakhter

aakhter commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, good catch, and thanks for the precise stacking analysis. Fixed in ee1a155, going with your first option.

  • The overlay now draws the composition. ZerolagInputAddon.setComposition(text) in packages/xterm-zerolag-input/src/ renders it as an underlined tail after pendingText, reusing the existing wrapping and grow-upward layout. It never enters pendingText, hasPending or state, so it can never be flushed. clear() (Enter, Ctrl+C) and removeChar() drop it; rerender() and refreshFont() keep it. The spans are aria-hidden. The package has 15 new unit tests.
  • terminal-ui.js routing. With local echo on, the preview renders and clears through setComposition. The helper span stays for local echo off (shell sessions, where the PTY cursor is in the right place), and as a fallback if the overlay can't find the prompt.
  • The missing test. test/mobile-ime-preview.browser.test.ts now loads real xterm 6, the overlay bundled from source and styles.css, commits 今日は into the overlay, then composes 天気. It checks that the composition is the topmost element after the pending text. On the old code it found the overlay's opaque line there, exactly as you described. Turning the routing off fails 2 of the browser tests.
  • CLAUDE.md and docs/architecture-invariants.md now state the effective layer. With local echo on, the preview is part of the overlay at z-index 7. With it off, it is the helper span at 6 inside .xterm-helpers, so effectively 5.

Looking forward to your iPhone run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants